-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-6314][CORE] handle JsonParseException for history server #5736
Conversation
Test build #31121 has finished for PR 5736 at commit
|
That's probably fine. CC @andrewor14 @vanzin |
@@ -333,8 +333,9 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis | |||
} | |||
try { | |||
val appListener = new ApplicationEventListener | |||
val maybeTruncated = logPath.getName().endsWith(EventLoggingListener.IN_PROGRESS) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to call isApplicationCompleted(eventLog)
instead, to handle legacy logs. (You can then replace the same call a few lines below with the result, to avoid another trip to HDFS.)
LGTM aside from the one issue I pointed out. |
This is handled in the same way with [SPARK-6197](https://issues.apache.org/jira/browse/SPARK-6197). The result of this PR is that exception showed in history server log will be replaced by a warning, and the application that with un-complete history log file will be listed on history server webUI Author: Zhang, Liye <liye.zhang@intel.com> Closes #5736 from liyezhang556520/SPARK-6314 and squashes the following commits: b8d2d88 [Zhang, Liye] handle JsonParseException for history server
I merged this after addressing @vanzin's comments myself into master and 1.3. |
Thanks to @andrewor14 and @vanzin's comments |
This is handled in the same way with [SPARK-6197](https://issues.apache.org/jira/browse/SPARK-6197). The result of this PR is that exception showed in history server log will be replaced by a warning, and the application that with un-complete history log file will be listed on history server webUI Author: Zhang, Liye <liye.zhang@intel.com> Closes apache#5736 from liyezhang556520/SPARK-6314 and squashes the following commits: b8d2d88 [Zhang, Liye] handle JsonParseException for history server
This is handled in the same way with [SPARK-6197](https://issues.apache.org/jira/browse/SPARK-6197). The result of this PR is that exception showed in history server log will be replaced by a warning, and the application that with un-complete history log file will be listed on history server webUI Author: Zhang, Liye <liye.zhang@intel.com> Closes apache#5736 from liyezhang556520/SPARK-6314 and squashes the following commits: b8d2d88 [Zhang, Liye] handle JsonParseException for history server
This is handled in the same way with SPARK-6197. The result of this PR is that exception showed in history server log will be replaced by a warning, and the application that with un-complete history log file will be listed on history server webUI